1. Field
The present discovery and invention relate generally to graphical user interfaces for computer systems and relate in particular to graphical user interfaces for special computer vision systems, sometimes and herein known as xe2x80x9cAugmented Reality(trademark)xe2x80x9d computer vision systems. Graphical user interfaces of the invention find great novelty in their interaction, responsiveness and function related to these highly specialized computer vision systems.
A clear and complete description of computer vision systems has been disclosed as U.S. pending patent application having a Ser. No. 08/119,360. In addition, some basic and preliminary description of graphical user interfaces as they may particularly relate to computer vision systems appears in the disclosure U.S. pending patent application having a Ser. No. 08/307,360. Further, other concepts and ideas relating to graphical user interfaces, were presented in disclosure U.S. pending patent application having a Ser. No. 08/411,299. Each of those three pending U.S. patent applications is believed to contain considerably useful information as it may relate to the present invention. Accordingly, each of those documents is incorporated herein this disclosure, by reference thereto.
In agreement with provisions of 35 U.S.C. xc2xa7120, this application is a Continuation-in-Part type application. It continues from Ser. No. 08/411,299, filed Mar. 27, 1995; which in turn continues from Ser. No. 08/119,360, filed Sep. 10, 1993, both remain pending as of this even date.
2. Prior Art
A graphical user interface is a computer generated graphical device which a computer user may employ to interact with, or command, a computer system to take some action or actions. A commonly recognized graphical user interface is known quite well to most computer users as a xe2x80x9cMenuxe2x80x9d. One example of a Menu includes a list of option selections presented in a simple list box. A user may select an option by pointing a cursor to it via a pointing device. Some pointing devices include: a mouse, a trackball, and scrolling keys or other tactile means. Pressing xe2x80x9centerxe2x80x9d or xe2x80x9cclickingxe2x80x9d a mouse button while a cursor is pointing to a selection then commands the computer to execute a function associated with the option selected.
Various types of Menus have been configured to interact with a user in different ways. Sometimes, and depending upon the application being run on a computer, one type of Menu may provide better function than another type. Two common types are discussed here to illustrate how a graphical user interface may preferentially interact with a particular application. A xe2x80x9cpop-upxe2x80x9d type Menu and a xe2x80x9cdrop-downxe2x80x9d type Menu each act differently; each having certain cooperation with respect to the application which the computer is running.
A xe2x80x9cpop-upxe2x80x9d type Menu may be initiated by some event in a computer program. It typically interrupts normal program activity. For example, if a computer error occurs, a xe2x80x9cpop-upxe2x80x9d Menu may appear in the middle of a display screen and offer a user the options: xe2x80x9ccontinuexe2x80x9d or xe2x80x9cstart overxe2x80x9d.
In comparison, a xe2x80x9cdrop-downxe2x80x9d Menu is typically initiated by request of a user. For example, an icon on a xe2x80x9ctool barxe2x80x9d may indicate a group of tasks related to a common feature. Stimulating (xe2x80x9cpointing and clickingxe2x80x9d) the icon causes a Menu box to drop down therefrom and into the display area. The Menu may have a list of possible command options which are selectable by a user. xe2x80x9cPop-upxe2x80x9d type Menus, therefore, cooperate better with internal or automatic mechanisms which may initiate them and xe2x80x9cdrop-downxe2x80x9d Menus may be better suited for functions which are initiated by a user. These are only a few of the many features well known in the arts of computer graphical user interface design.
Sometimes an application which a computer is running suggests a certain type of graphical user interfaces. Very elegantly designed xe2x80x9cdrop-downxe2x80x9d Menus having advanced features are used with sophisticated drawing programs. Examples which thoroughly illustrate this are the Menus employed by the CorelDRAW!(trademark) drawing software packages. Those having experience with advanced drawing software packages will appreciate how clever Menu configuration may greatly enhance the ease-of-use and efficiency of the application.
There exists many fundamental differences between the display of a simple personal computer and the display of a computer vision system. A computer vision system may employ an electronic camera and a computer graphics generator to formulate augmented images of real scenes in real-time. Composite images presented at the display of a computer vision system may be comprised of optically acquired images having been modified or augmented with computer generated graphics. The computer generated graphics may relate to objects detected (or otherwise xe2x80x9cknownxe2x80x9d by the computer) in the scene being addressed In particular, some objects are identified by their known location. The objects may be graphically simulated by, and superimposed onto xe2x80x9crealxe2x80x9d or optically acquired images of the objects.
It may be desirable for the user to command a computer vision system to perform various functions. Standard Menus, or other graphical user interfaces, can be employed by computer vision systems to provide for user interface function. However, since computer vision systems behave very differently than common computer systems, Menus which might be most useful in computer vision systems are heretofore completely unknown.
Particular function and features associated with computer vision systems which are not found in common computer systems suggest graphical user interfaces may be uniquely arranged to cooperate with those functions and features particular to those specialized systems. The present inventors have now discovered some very useful and valuable configurations of graphical user interfaces as they may particularly apply to computer vision systems. These new graphical user interfaces provide surprising results when considering the benefits they may provide to users of computer vision systems which employ them. The new graphical user interfaces tend to facilitate operation, enhance functionality, improve interpretation of images, increase understanding of scenes. These graphical user interfaces operate in a way which is not and cannot be used with prior systems.
A graphical user interface system has been invented to interact with features and function which are particular to computer vision systems.
Computer vision systems having augmented images may have a graphical user interface configured to appear to interact with real objects of a scene. A graphical user interface may be arranged to interact with the pointing direction of the computer vision system. Graphical user interfaces may be responsive to position and/or attitude as determined by the computer vision system. Graphical user interfaces may be responsive to a cursor which corresponds to a camera boresight indicator. Many configurations of graphical user interfaces which are particular to computer vision systems exist. These are presented in detail in the sections here following. When a computer vision system includes graphical user interface devices, the interaction of the graphical user interfaces with respect to elements of the system can produce some surprising results. The present invention is concerned with how graphical user interfaces may be arranged to interact with computer vision systems and elements thereof.
A fundamental difference between a simple computer and a computer vision system is that displayed images in the computer vision system correspond directly to some view of the real world. Images are aligned to the scene in real time. A computer vision system addresses a scene as its camera axis is pointing toward it. The computer vision system has associated with it at all times, a position and attitude which are easily measurable and thus xe2x80x9cknownxe2x80x9d to the computer. As a result, displayed images are presented in a certain perspective which corresponds to the point-of-view of the computer vision system and the user""s position. The displayed images of a simple computer are not generally associated with its surrounding environment nor aligned to any scene.
Besides using the computer vision system pointing direction as an interface pointer, a graphical user interface may be arranged to respond to the pointing direction of the computer vision system when the boresight is not acting as a pointer. A graphical user interface might respond to the absolute pointing direction.
The xe2x80x9cpointing directionxe2x80x9d of a computer vision system is a primary feature which should be well understood. The pointing direction will be shown to have great interaction with various graphical user interfaces. Not only does the pointing direction sometimes serve as an interface pointing cursor but the absolute pointing direction may influence the behavior of certain graphical user interfaces.
graphical user interfaces of the invention can be made to be responsive to a new type cursor or xe2x80x9cpointerxe2x80x9d. Operations known as xe2x80x9cpoint-and-clickxe2x80x9d and xe2x80x9cdrag-and-dropxe2x80x9d were heretofore performed with a computer peripheral pointer known as a xe2x80x9cmousexe2x80x9d, xe2x80x9ctrack ballxe2x80x9d, or xe2x80x9cpowerpointxe2x80x9d. Each of these devices allow a user to cause a pointing icon or xe2x80x9cpointerxe2x80x9d to traverse a display field. When the icon is collocated with something of interest in the displayed image, an object, image or another icon for example, then some action can be initiated by the computer program. The pointer of a computer vision system may include a conventional pointer which moves in response to a tactile stimulus, or might by arranged such that it corresponds to the vision system boresight. By pointing the computer vision system in any direction, a user causes the boresight to be collocated with some part of the displayed image. By pointing the computer vision system at objects of interest, the use might indicate commands to the computer. Use of a vision system boresight as a pointer is believed to be a completely new concept.
For example, a drop-down Menu may be associated or xe2x80x9cattachedxe2x80x9d to some object of the scene. The position of the graphical user interface is made to correspond at all times with the apparent position of the object. When the pointing direction of the vision system is panned across a horizon, objects in the scene appear on a display to move laterally. A Menu associated with a particular object can be made to appear to follow the object""s lateral motion so that the graphical user interface stays with the object to which it is attached. Even if the object leaves the field-of-view of the vision system, so can the drop-down Menu. When the object is again acquired by the system (comes again into the field-of-view), then so does the drop down Menu. This example illustrates that a graphical user interface may interact in real time with the pointing direction of the computer vision system. i.e. when the pointing direction is adjusted, the graphical user interface responds to the adjustment.
In addition, the absolute location of the computer vision system may dictate the behavior of certain graphical user interfaces. If the computer vision system is in Paris, graphical user interfaces may automatically be presented in the French language. Similarly, if the computer vision system is determined to be in New York, then graphical user interfaces may automatically be presented in the English language. It is quite possible that the combination of both position and attitude of the computer vision system may affect the behavior of graphical user interfaces. It is further possible that the display field periphery may be enabled such that it might operate on a graphical user interface. Other features and function particular to computer vision systems can be exploited to advance the usefulness and suggest arrangements of graphical user interfaces for computer vision systems. The example above shows how pointing direction might affect a graphical user interface, some further examples directed to position, position and attitude, magnification are briefly mentioned here. Some specific examples follow.
Position
A simple example shows how position alone might dictate the graphical user interface content and behavior of a graphical user interface. Since a graphical user interface may be an icon of arbitrary shape, it is possible that a small map in the shape of a state""s boundary be displayed with images of scenes being addressed. As the computer vision system is moved from state-to-state, the map could change shape to correspond to the state that it is in. As a user crosses the border from Kansas City, Miss. to Kansas City, Kans., a Missouri shaped icon would become a Kansas shaped icon.
Position and Attitude
Under some circumstances, both position and attitude of a computer vision system are used to create a special version of a graphical user interface. If a scene includes a particular object for example a billboard, then the billboard will appear in a different perspective for every location from which it may be viewed. It may be desirable to have a graphical user interface appear in the same perspective as the billboard. To accomplish this, a determination of the computer vision system position and attitude enables the computer to compute the proper perspective associated with any place from which the billboard may be viewed or addressed by the system. The graphical user interface can then be displayed in a perspective which corresponds to the perspective of the billboard. Thus, both position and attitude of the computer vision system may affect the arrangement of a graphical user interface of the invention.
Magnification
Computer vision systems have very sophisticated zoom properties, graphical user interfaces of computer vision systems may aid in serving zoom objectives of those systems. A graphical user interface may be arranged such that its content may include magnified images of scenes being addressed, while the display field contains a non-magnified image.
From the Image Signal
The electronic image signal generated by the camera may be probed by the computer to detect some feature. From this information a graphical user interface may be generated to aid in understanding and interacting with the detected feature.
To fully appreciate the invention, one should have a complete understanding of computer vision systems of the type which produce augmented images of real scenes. Full disclosure of those systems has been made, referenced and incorporated herein this document. A brief review follows; however, one cannot be expected to completely understand this disclosure without full understanding of the references as significant elements herein are defined at length in those presentations. Therefore, thorough review of the incorporated documents is highly recommended.
Computer vision systems may be comprised of a camera, a computer, a display and position and attitude determining means. The system addresses a scene and produces at the display an augmented image thereof. The augmented image is comprised of image information from two sources. A first image source is the camera which optically acquires an image of a scene and produces an electronic image signal. The second source is a computer generated image source. From position and attitude measurements, the computer identifies a scene and objects therein. The computer may recall from memory, models which are related to identified objects and assembles a computer generated image having a perspective which matches the true perspective of the scene from the point-of-view of the computer vision system in real time. Optically acquired images are combined with computer generated images to form composite or xe2x80x9caugmentedxe2x80x9d images. An augmented image is presented to a user on a display having a display field aligned to the scene being addressed. A user views the xe2x80x9crealxe2x80x9d world where the display of the computer vision system appears to be a xe2x80x9cwindowxe2x80x9d at which the user looks. A user finds great benefit because the xe2x80x9cwindowxe2x80x9d may contain image information about the real world in true scale and proper perspective. This information may additionally contain objects which are not readily visible to the naked eye or the electronic camera as the computer generates portions of the final image.
It is a primary object of the invention to provide graphical interface to computer systems.
It is a further object to provide graphical user interfaces to computer vision systems.
It is still further an object to provide new graphical user interfaces.
It is an object of the invention to provide new function to graphical user interfaces.
It is an object to provide graphical user interfaces which are responsive to objects and features in augmented images.
It is an object to provide graphical user interfaces which are responsive to computer vision system functions, features, and properties.
It is an object to provide graphical user interfaces which are responsive to position or attitude, or both, of computer vision systems.
In accordance with these objectives, certain preferred examples including the best modes anticipated are presented herefollowing in great detail with clear description having references to drawing figures.